The following methodology and function parameters are based directly on Jenna Vergeynst’s Python code.
Highlighted blocks are quotes from Jenna’s repository.

Use this code to synchronize receivers based on detection times of synchronization transmitters (‘sync-tags’).
The synchronized times can then be processed into a TOA matrix and fed into YAPS to get estimated locations.

Note that the following synchronization process assumes that the sync-tags are embedded within their resepctive receivers (i.e., are not spatially separated).

Synchronization overview

If possible, apply linear corrections to the receives (e.g., VEMCO’s time drift Auto Correct option in VUE)
Choose one receiver as the “base receiver”, preferably a receiver in the middle of the array, that hears all of the other receivers. The base receiver is the time keeper, the clock of the other receivers will be synchronised with the base clock.
Go through the synchronisation process for each receiver. In this process, you compare the detection time of 2 receivers in a receiver pair (base receiver versus receiver in process) detecting the same transmitter (synchronisation transmitter of one of the pair). The goal is to model the spline: a function that reflects the deviation of the receiver clock from the base clock over time.
Check the figures and play with the parameters of the functions to get a smooth spline that fits (but NOT overfits!) the DTDs as good as possible.
If you have receivers without synchronisation transmitter, it is still possible to synchronise them:
- Model the spline of the DTD of the base receiver synchronisation transmitter between both receivers
- Convert the distance between both receivers in time-distance by use of soundspeed
- For the resulting spline, substract the time-distance from the base transmitter spline.

Setup

The pkg_run function will load required libraries (and versions), installing/updating them first as needed

# Disable scientific notation
options(scipen=999) 

# Print microseconds when viewing the data
sigFiguresSec = 20
options(digits.secs=sigFiguresSec)

# Import custom R functions
 for (f in list.files('../code', pattern='*.R')) {
     source(paste0('../code/', f))
 }

pkg_run(list(
            'lubridate', 
            c('data.table', '1.12.0'),
            'dygraphs'),
       load_verbose = FALSE,
       inst_verbose = TRUE)

Load data

Detection times from each receiver should be in their own file.
Supply the timezone for the datetime column

The following sample data for base_rec and other_rec were converted from pickle into csv.

base_rec_path = '../data/example_base_461059_jv.csv'
other_rec_path = '../data/example_rec_461211_jv.csv'

data_tz = 'GMT'
# same as data_tz = 'UTC'

Sys.setenv(TZ = data_tz)

Read the df’s and configure the datetime column

base_receiver = fread(base_rec_path, sep = ',', header = TRUE)[, 
                Time := ymd_hms(Time, tz = data_tz)]

other_receiver = fread(other_rec_path, sep = ',', header = TRUE)[, 
                 Time := ymd_hms(Time, tz = data_tz)]

Total rows base_receiver: 2,262,484
Total rows other_receiver: 2,163,807

head(other_receiver)

##                          Time    ID
## 1: 2018-05-01 00:00:03.481616 65001
## 2: 2018-05-01 00:00:06.686088 62209
## 3: 2018-05-01 00:00:08.906914 62215
## 4: 2018-05-01 00:00:10.483454 62214
## 5: 2018-05-01 00:00:13.689360 62211
## 6: 2018-05-01 00:00:13.776442 62059

Specify the sync ID’s for the base rec and the other rec (bsync_id and osync_id, respectively)
Specify the datetime and transmitter ID column names in the df’s

bsync_id = '62059'
osync_id = '62211'

dt_col = 'Time'
id_col = 'ID'

DTD and spline of other rec sync-tag

Add DTD

Subset base_receiver and other_receiver df’s to the detections of the other_rec sync-tag.
Then, for each detection in the other_receiver sub df, find the nearest detection (within time = t_sec seconds) in the base_receiver sub df
Calculate DTD: the absolute time difference between each detection pair

orec_osync = add_dtd(b_rec_data = base_receiver,
                     o_rec_data = other_receiver,
                     sync_id = osync_id,
                     dt_cols = 'Time',
                     id_cols = 'ID',
                     t_sec = 100)

Chunk execution time: 0.97 sec

Plot the DTD
Adjust plot_subset in order to plot only a subset of the of data, which is useful for large datasets (e.g., plot_subset = 3 plots every 3rd value). Disable this feature by setting to FALSE (or 1).

dg_plot(orec_osync, dt_col = dt_col, plot_subset = 6, 
        main = 'DTD of other rec sync-tag')

Smooth DTD

Smooth the DTD to avoid a subsequent jumpy spline
Play with the parameters to get the DTD smooth enough (BUT it does not have to be entirely smooth => the spline can handle some noisiness).
e.g. in the example, if you set window_size=1 with outlier_lim=0.005, you will see the effect of insufficient smoothing
Zoom in on the figure to check smoothness! (especially at jumps!)
NOTE: if you smooth less here, the parameter s for modelling the spline will probably have to be larger (in some case this is the best solution).

smooth_dtd(orec_osync,
           outlier_lim = 0.005, 
           window_size = 5)

Chunk execution time: 0.03 sec

dg_plot(orec_osync, dt_col = dt_col, plot_subset = 6, line_w = 0.7, 
        main = 'Smooth DTD of other rec sync-tag')

# If the original, un-smoothed DTD appears smooth enough, uncomment and run the following:
# orec_osync[, dtd_smooth := dtd]

Spline DTD

Split the df at large DTD gaps (> gap_lim sec)
For each split, model the DTD spline

NOTE: The spline method (and the k_par and s_par paramters) used here (smooth.spline) is different from the one in Jenna’s code (Scipy’s UnivariateSpline), but results should be fairly similar

spline_groups = model_spline_part(orec_osync, target_data = other_receiver, dt_cols = dt_col, 
                                  gap_lim = 0.1, k_par = 5, s_par = 5e-4)

Chunk execution time: 1.95 sec

Plot the resulting DTD spline
Check if there is nice correspondance between spline and DTD
Zoom in closely to see this well, especially on break points.
If there is no nice correspondance, play with the parameters of model_spline_part or smooth_DTD.
For instance:
- if overshoot of the spline at the breaks: you probably need to make the DTD smoother => decrease outlier_lim and/or increase window_size
- if discrepancy between spline and DTD line is too large => decrease window_size (can even be 1)
- if some part is not modelled: probably because it was lost by smoothing => increase outlier_lim
- if the spline is overpredicting (i.e. follows every tiniest variation in DTD_smooth) => increase s and/or decrease k

dg_plot(merge(x = spline_groups, y = orec_osync, all = TRUE), 
        dt_col = dt_col,
        y_cols = c('spline', 'dtd_smooth'),
        main = 'DTD spline of other rec sync-tag',
        plot_subset = 20, 
        line_w = 0.7)

Add the spline to the original other_recevier df

other_receiver[spline_groups, on = dt_col, spline_osync := i.spline]

DTD and spline of base rec sync-tag

Repeat the previous process for the detections of the base receiver sync-tag

Add DTD

orec_bsync = add_dtd(b_rec_data = base_receiver,
                     o_rec_data = other_receiver,
                     sync_id = bsync_id,
                     dt_cols = 'Time',
                     id_cols = 'ID',
                     t_sec = 100)

Chunk execution time: 1.16 sec

dg_plot(orec_bsync, dt_col = dt_col, plot_subset = 6, line_w = 1, 
        main = 'DTD of base rec sync-tag')

Smooth DTD

smooth_dtd(orec_bsync, 
           outlier_lim = 0.001, 
           window_size = 6)

Chunk execution time: 0.02 sec

dg_plot(orec_bsync, dt_col = dt_col, plot_subset = 6, line_w = 1, 
        main = 'Smooth DTD of base rec sync-tag')

# If the original, un-smoothed DTD appears smooth enough, uncomment and run the following:
#orec_bsync[, dtd_smooth := dtd]

Spline DTD

spline_groups = model_spline_part(orec_bsync, target_data = other_receiver, dt_cols = dt_col, k_par = 5, s_par = 5e-4)

Chunk execution time: 1.09 sec

dg_plot(merge(x = spline_groups, y = orec_bsync, all = TRUE), 
        dt_col = dt_col, 
        y_cols = c('spline', 'dtd_smooth'),
        main = 'DTD spline of base rec sync-tag', 
        plot_subset = 50, 
        line_w = 0.7)

Add the spline to the original other_recevier df

setkeyv(other_receiver, dt_col)[spline_groups, on = dt_col, spline_bsync := i.spline]

Chunk execution time: 0.53 sec

Combine splines

Calculate the average spline from the splines of the two sync-tags
Fill missing spline values by linear interpolation

other_receiver[, spline := (spline_bsync + spline_osync) / 2] %>% 
    dt_linear_interp(dt_col, 'spline')

Chunk execution time: 0.31 sec

dg_plot(other_receiver, dt_col = dt_col, 
        y_cols = c('spline_bsync', 'spline_osync', 'spline'),
        main = 'DTD Spline of the other rec vs base rec',
        plot_subset = 50, 
        line_w = 0.7)

other_receiver[, synced_time := (get(dt_col) - spline)]

Check DTD after synchronisation

Re-run the the process of adding DTD’s with the newly synchronized times
Synchronized detection times should center around zero

orec_osync = add_dtd(b_rec_data = base_receiver,
                     o_rec_data = other_receiver,
                     sync_id = osync_id,
                     dt_cols = c(dt_col, 'synced_time'),
                     t_sec = 100)

smooth_dtd(orec_osync, 
           outlier_lim = 0.001, 
           window_size = 6)

orec_bsync = add_dtd(b_rec_data = base_receiver,
                     o_rec_data = other_receiver,
                     sync_id = bsync_id,
                     dt_cols = c(dt_col, 'synced_time'),
                     t_sec = 100)

smooth_dtd(orec_bsync,
           outlier_lim = 0.001, 
           window_size = 6)

Chunk execution time: 2.19 sec

dg_plot(merge(x = orec_osync, y = orec_bsync, all = TRUE), 
        dt_col = 'synced_time', 
        y_cols = c('dtd_smooth.x', 'dtd_smooth.y'),
        plot_subset = 20, line_w = 0.7)

Session Info

sessionInfo()

## R version 3.6.0 (2019-04-26)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS High Sierra 10.13.6
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] dygraphs_1.1.1.6  data.table_1.12.2 lubridate_1.7.4  
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.1      lattice_0.20-38 zoo_1.8-5       digest_0.6.18  
##  [5] grid_3.6.0      jsonlite_1.6    magrittr_1.5    evaluate_0.13  
##  [9] stringi_1.4.3   xts_0.11-2      rmarkdown_1.12  tools_3.6.0    
## [13] stringr_1.4.0   htmlwidgets_1.3 xfun_0.6        yaml_2.2.0     
## [17] parallel_3.6.0  compiler_3.6.0  htmltools_0.3.6 knitr_1.22

Receiver array time synchronization using splines and sync-tags

Synchronization overview

Setup

Load data

DTD and spline of other rec sync-tag

Add DTD

Chunk execution time: 0.97 sec

Smooth DTD

Chunk execution time: 0.03 sec

Spline DTD

Chunk execution time: 1.95 sec

DTD and spline of base rec sync-tag

Add DTD

Chunk execution time: 1.16 sec

Smooth DTD

Chunk execution time: 0.02 sec

Spline DTD

Chunk execution time: 1.09 sec

Chunk execution time: 0.53 sec

Combine splines

Chunk execution time: 0.31 sec

Check DTD after synchronisation

Chunk execution time: 2.19 sec

Session Info